ENUMERATION OF STANDARD YOUNG TABLEAUX 
OF CERTAIN TRUNCATED SHAPES 



RON M. ADIN, RONALD C. KING, AND YUVAL ROICHMAN 

Abstract. Unexpected product formulas for the number of standard Young 
tableaux of certain truncated shapes are found and proved. These include 
shifted staircase shapes minus a square in the NE corner, rectangular shapes 
minus a square in the NE corner, and some variations. 



1. Introduction 

A truncated shape is obtained from a Ferrers diagram (in the English notation, 
where parts decrease from top to bottom) by deleting cells from the NE corner. 
Interest in the enumeration of standard Young tableaux of truncated shapes is 
enhanced by a recent result [1] Prop. 9.7]: the number of geodesies between 
distinguished pairs of antipodes in the flip graph of triangle-free triangulations 
is equal to twice the number of Young tableaux of a truncated shifted staircase 
shape. Motivated by this result, extensive computations were carried out for 
the number of standard Young tableaux of these and other truncated shapes. It 
was found that, in certain distinguished cases, all prime factors of these numbers 
are relatively small, hinting at the existence of product formulas. In this paper, 
such product formulas are proved for rectangular and shifted staircase shapes 
truncated by a square, or nearly a square. 

A different method to derive product formulas, for other families of truncated 
shapes, has independently been developed by G. Panova [TJ. 

The rest of the paper is organized as follows. Basic concepts are described in 
Section [5J The general idea of pivoting is presented in Section [3] Section Q] con- 
tains detailed proofs for truncated shifted staircase shapes, with Theorem 14.61 as 
the main result; while Section [5] contains an analogous development for truncated 
rectangular shapes, with Theorem l5.5l as the main result. Section[6]contains final 
remarks and open problems. 
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2. Preliminaries and Basic Concepts 

A partition A of a positive integer N is a sequence of non-negative integers 
(Ai, A2, . . • , A m ) such that Ai > A2 > . . . > A m > and the total size |A| := 
Ai + A2 + . . . + A TO is N. The Ferrers diagram [A] of shape A is a left-justified 
array of N cells, with row i (from top to bottom) containing Ai cells. A standard 
Young tableau (SYT) T of shape A is a labeling by {1, 2, ... , N} of the cells in the 
diagram [A] such that every row is increasing from left to right, and every column 
is increasing from top to bottom. The number of SYT of shape A is denoted by 
f X - 

Proposition 2.1. (The Frobenius- Young Formula) [HUH] The number of SYT 
of shape A = (Ai, A2, . . . , A m ) with Ai > A2 > . . . > A m > is 

Note that adding trailing zeros to A (with m appropriately increased) does not 
affect the right-hand side of the above formula. 

A partition (Ai, A2, . . . , A m ) of N is strict if Ai > A2 > . . . > A TO > 0. The 
corresponding diagram of shifted shape A is the array of N cells with row i con- 
taining Ai cells and indented i — 1 places. A standard Young tableau (SYT) T of 
shifted shape A is a labeling by {1,2,..., N} of the cells in the diagram [A] such 
that each row and column is increasing. The number of SYT of shifted shape A 
is denoted by g x . 

Proposition 2.2. (Schur's Formula) [5] p. 267 (2)] The number of SYT of 
shifted shape A = (Ai, A2, . . . , A m ) with Ai > A2 > . . . > A m > is 

x = ML TT Xi Aj 

m -I- -I- A -4- A ' 

It is well known that both / A and g x also have hook length formulas, but the 
equivalent formulas above will be more convenient for our calculations. 

We shall frequently use the following two basic operations on partitions. The 
union A U /1 of two partitions A and \i is simply their multiset union. We shall 
usually assume that each part of A is greater or equal than each part of /1. The 
sum of two partitions A = (Ai, . . . , A TO ) and \i = (//i, . . . , /i m ) (with trailing zeros 
added in order to get the same number of parts) is 

X + fi:= (Xl+fx 1 ,...,X m + fj, m ). 

For any nonnegative integer m, let [m] := (m, m— 1, . . . , 1) be the correspond- 
ing shifted staircase shape. Consider the truncated shifted staircase shape [m] \k, 
where a partition k = (ki, Ku), with m < m — i for all 1 < i < k < m, is 
deleted from the NE corner; namely, n\ cells are deleted from the (right) end 
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of the first row, K2 cells are deleted from the end of the second row, etc. Let 
N be the size of [m] \ k. A standard Young tableau (SYT) of truncated shifted 
staircase shape [m] \ k is a labeling of the cells of this shape by {1, . . . , N} such 
that labels increase across rows (from left to right), down columns (from top to 
bottom) and down the main diagonal (from top left to bottom right). 

Example 2.3. There are four SYT of shape [4] \ (1): 




Similarly, for nonnegative integers m and n, let (n m ) — {n, . . . , n) (m parts) be 
the corresponding rectangular shape. Consider the truncated rectangular shape 
(n m ) \ k, where k C (n m ) is deleted from the NE corner; namely, ki cells are 
deleted from the end of the first row, K2 cells are deleted from the end of the 
second row, etc. Letting N be the size of (n™ 1 ) \ k, a SYT of truncated shape 
(n m ) \ k is a labeling of the cells of this shape by {1, ... , N}, such that labels 
increase along rows (from left to right) and down columns (from top to bottom). 

Preliminary computer experiments hinted that a remarkable phenomenon oc- 
curs when a square is truncated from a staircase shape: while the number of SYT 
of truncated staircase shape [m] \ (k k ) increases exponentially as a function of 
the size N = C™^ 1 ) — k 2 of the shape, all the prime factors of this number are 
actually smaller than the size. A similar phenomenon occurs for squares trun- 
cated from rectangular shapes 0. In this paper, product formulas for these (and 
related) truncated shapes, explaining the above factorization phenomenon, will 
be proved; see Corollaries 14.71 14. 8[ 15.61 and 15.71 below. 

3. Pivoting 

The basic idea of the proofs in the following sections is to enumerate the SYT 
T of a given shape £ by mapping them bijectively to pairs (T\,T2) of SYT of 
some other shapes. This will be done in two distinct (but superficially similar) 
ways, which will complement each other and lead to the desired results. 

The first bijection will be applied only to non-truncated shapes of two types: 
a rectangle and a shifted staircase. Let N be the size of the shape ( (of either of 
the aforementioned types), and fix an integer I < t < N. Subdivide the entries 
in a SYT T of shape £ into those that are less than or equal to t and those that 
are greater than t. The entries < t constitute T\. To obtain T2, replace each 



It should be noted that this factorization phenomenon does not hold in the general case. 
For example, the number of SYT of truncated rectangular shape (7 6 ) \ (2), of size 40, has 5333 
as a prime factor. 
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entry i > t of T by N — i + l, and suitably transpose (actually, reflect in the line 
y = x) the resulting array. It is easy to see that both T\ and T 2 are SYT. 

This is illustrated schematically in the case of a shifted staircase shape by the 
following shifted SYT, where C = (5, 4, 3, 2, 1), N = 15 and t = 7. 




4^ 



7i,T 2 




In terms of shapes we have 



[C] 



where r' denotes the conjugate of the (strict) partition r. It follows immediately 
from this argument that 



E 

<tCC 



g" g 



(CM' 



where r = (C/c)' is the conjugate of the shifted skew shape (/a. The shape [t] 
is formed by deleting the cells of [cr] from those of [Q and then reflecting in the 
line y = x. 

In the case of a rectangular shape we have 



[C] 



with 



The second bijection will be applied to truncated shapes, where a partition is 
truncated from the NE corner of either a rectangle or a shifted staircase. Given 
such a truncated shape £, choose a pivot cell P. This is a cell of ( which belongs 
to its NE boundary, namely such that there is no cell of £ which is both strictly 
north and strictly east of the cell P. If T is a SYT of shape £, let t be the entry 
of T in the pivot cell P. Subdivide the other entries of T into those that are 
(strictly) less than t and those that are greater than t. The entries less than t 
constitute T\. To obtain T 2 , replace each entry i > t of T by N — i + 1, where N 
is the total number of entries in T, and suitably transpose the resulting array. It 
is easy, once again, to see that both T\ and T 2 are SYT. 

By way of example, consider the truncated rectangular shape specified by 
C = (4,5,7,8,8) and let P be the cell in position (3,5). Then for t = 17 the 
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map from a SYT T of truncated shape [£] to a corresponding pair (T\,T2) is 
illustrated by 



1 2 4 9 | 

3 5 11 12 13 
6 8 14 lb\r7 



21 24 



3i,T 2 



7 16J18 20 25 26 27 30 
10 [l9 22 23 28 29 31 32 



1 2 4 9 | 

3 5 11 12^j] 
6 8 14 15| 
T 16T 

iof 



1 3L 

2 6 9 

4 7 12 

5 8 

10 13 

11 15 
I4I 



In terms of shapes we have 



where r' is the conjugate of r. Similarly for a truncated shifted staircase shape. 
The particular shapes required in the sequel are the following: 



(1) 



and 



(2) 



fi+a 











M + Q 





















A crucial property of these particular shapes is that their subdivision gives rise 
to shapes [i U A and fx U v (or {jx + a) U A and (/i + /3) U f) which are noi truncated. 

A more explicit relation between A and v will be given later. For the time 
being it suffices to observe that 



(3) 



XC[m] 
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and 

(4) fC = /(M+")UA y( M +/3)u((m")/A)' _ 

AC(ri m ) 

Since C is a truncated shape, the notation on the LHS of the above equalities is 
to be taken in a generalized sense. 

4. Truncated Shifted Staircase Shapes 

In this section, A = (Ai, . . . , A m ) (with Ai > . . . > A m > integers) will be 
a strict partition, with g x denoting the number of SYT of shifted shape A. We 
shall use the union operation on partitions, defined in Section [2] 

For any nonnegative integer to, let [to] := (to, to — 1, . . . , 1) be the corre- 
sponding shifted staircase shape. Schur's formula (Proposition 12. 2[) implies the 
following. 

Observation 4.1. The number of SYT of shifted staircase shape [to] is 

m— 1 .■ 

g [m] = mi . TT 

y 11 (2i+l) 

where M := \[m]\ = ( TO + X ). 

We shall now use the first bijcction outlined in Section [3] 

Lemma 4.2. Let m and t be nonnegative integers, with t < ( m ^ 1 )- Let T be a 
SYT of shifted staircase shape [to], let T\ be the set of all cells in T with values at 
most t, and let T2 be obtained from T\T\ by transposing the shape and replacing 
each entry i by M — i + 1. Then: 

(i) Ti and T2 are shifted SYT. 

(ii) Treating strict partitions as sets, [to] is the disjoint union of the shape of 
T\ and the the shape of T2 . 

Proof, (i) is clear. In order to prove (ii), denote the shifted shapes of T\ and 
T2 by A 1 and A 2 , respectively. The borderline between T\ and T\T\ is a lattice 
path of length exactly to, starting at the NE corner of the staircase shape [to] and 
using only S and W steps. If the first step is S then the first part of A 1 is to, and 
the rest (of both A 1 and A 2 ) corresponds to a lattice path in [to — 1]. Similarly, 
if the first step is W then the first part of A 2 is to, and the rest corresponds to 
a lattice path in [to — 1]. Thus exactly one of A 1 , A 2 has a part equal to to, and 
the whole result follows by induction on to. 

□ 

Corollary 4.3. For any nonnegative integers to and t with t < ( m ^~ 1 ), 
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Here summation is over all strict partitions X with the prescribed restrictions, 
and X c is the complement of X in [m] (where strict partitions are treated as sets). 
In particular, the LHS is independent oft. 

Lemma 4.4. Let X and X c be strict partitions whose disjoint union (as sets) is 
[m], and let fi = . . . with Hi > . . . > fi^ > m. Let g x := g A /|A|!. Then 

-flUX'HUX' ~[m] _ g X g X " gVU[™] (jV- _ 

Equivalently, 

3" U V UAC = c( M ,|A|,|A c |)- 9 V c , 

where 

< im ixcn - ^ uH ^ IH|!(H + |A|)!(|mI + |A c |)! 
1,1 U flH ' (H + |HI)!H!|A|!|A=|! 
depends only on the sizes \X\ and |A C | and not on the actual partitions X and X c . 

Proof. By Proposition 12.21 

W 9 X 9 XC \ 11 M,! y W + A*i J 1 1 W + Aj Mi + AS ' 

By the assumption on A and A°, 

\ \ c m 

3 3 3 3 = 1 

Thus the RHS of ([5]) is independent of A and A c . Substituting A = [m] (and 
X c = [0], the empty partition) yields 

g x g x " ~ §H ' 
which is the desired identity. The other equivalent formulation follows readily. 

□ 

A technical lemma, which will be used to prove Theorems 14.61 and 15.51 is the 
following. 

Lemma 4.5. Let t\, ti and N be nonnegative integers. Then 

N 

E 



i=0 



h + A (h + N - A = (h + * 2 + N + 1 

h ) \ h ) I t x + 1 2 + 1 



Proof. This is a classical binomial identity, which follows for example from 
computation of the coefficients of x N on both sides of the identity 

(i - x)-^ l+t ^ ■ (i - x)-^ l+t ^ = (i - x )-v +u+t2 \ 

□ 
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Theorem 4.6. Let m be a nonnegative integer, denote M :— C™^" 1 ), and let 
[i = (/ii, . . . , fj,k) be a strict partition with fi\ > . . . > fik > m. Then 

V „muA„muA c = „,uH« (M + 2H + 1)!H! 
J^i '(M + H)!(2| M | + l)r 

AC [m\ 

Proof. Restrict the summation on the LHS to strict partitions A of a fixed size 
A| = t. By Lemma Ol and Corollary Ol 

£ ^ u V UA ° = cGM,M-i)- =c(»,t,M-t)-gW. 

XC[m] AC[m] 
|A|=t |A|=t 

Now sum over all t and use the explicit formula for c(/z, t, M—t) (from Lemma [4.4[) 
together with Lemma T4. 5 1 

Acjm] + t\{M-t)\ 

\n\ + M\~ 1 ^f\n\+t\f\n\+M-t 



E 



|/i| + AA _1 /2|/i| + M + l 

ImI J A 2| M | + i 



□ 



We shall apply this theorem to several special cases. In each case the result 
will follow from an application of equation ([3]) to one or the other of the shapes 
illustrated in diagram ([lj, where the union of A and v is the shifted staircase 
shape [m]. Using Lemma l4.2| we can now state explicitly the relation between 
these partitions, mentioned before equation ([3]): v = (\m]/X)' = A c . 

First, take /j, = (m + k, . . . , m + 1) (k parts), for k > 1. This corresponds to 
truncating a k x k square from the NE corner of a shifted staircase shape [m + 2k], 
but adding back the SW corner of this square; see the first shape in diagram (Q]). 

Corollary 4.7. The number of SYT of truncated shifted staircase shape [m + 
2k] \ (k k -\k- 1) is 

g[m+k] g (m+k,...,m+l) j^|Mj 



(TV - | M i - 1)1(21^1 + 1)!' 
where N = ( m +^+i) -k 2 + l is the size of the shape and \fj,\ = k(2m + k + l)/2. 

The special case k = 1 (with [i = (m + 1)) gives back the number g[" l + 2 l of 
SYT of shifted staircase shape [m + 2] : 

q [m+l] N\(m + 1)1 _ tl [m+2] 

y (JV-m-2)!(2m + 3)! ' £1 (2< + l)! J 
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where N = (m + 2)(m + 3)/2 is the size of the shape. This agrees, of course, with 
Observation 14.11 

The special case k = 2 (with fi = (m + 2, m + 1)) corresponds to truncating a 
small shifted staircase shape [2] = (2, 1) from the shifted staircase shape [m + 4]. 
Thus, the number of SYT of truncated shifted staircase shape [m + 4] \ [2] is 

[m+2] (m+2,m+l) AH(2m + 3)! 

y y (iV-2rn-4)!(4m + 7)! 

yt 1 i\ (2m + 3)! AH(2m + 3)! 



i=0 



{2i + l)\ (m + 2)!(m+ l)!(2m + 3) (4m + 7)! 



AT! TT 

(Am 4- 7Vfm -A- 9^ 11 



(4m + 7)!(m + 2) l = l(2i + l)!' 



where N = ( m + 5 ) - 3 = (771 + 2)(m + 7)/2 is the size of the shape. 

Now take = (m + k + 1, . . . , m + 3, m + 1) (fc parts), for k > 2. This 
corresponds to truncating a (fc — 1) x (k — 1) square from the NE corner of a 
shifted staircase shape [m + 2k]; see the second shape in diagram ([T]). 

Corollary 4.8. The number of SYT of truncated shifted staircase shape [m + 
2k] \ ((k - l)^ 1 ) is 



g(m+k+l,...,m+3,m+l,...,l) g(m+k+l,...,m+3,m+l) 



iV!| M |! 



(JV-H-1)!(2| M | + 1)! ! 

where N — (J n+2 2 k+ls j — (k—1) 2 is the size of the shape and |^| = fc(2m+fc+3)/2— 1. 

In particular, take k = 2 and /i = (m + 3, m + 1). This corresponds to trun- 
cating the NE corner cell of a shifted staircase shape [m + 4] . The corresponding 
enumeration problem was actually the original motivation for the current work, 
because of its combinatorial interpretation, as explained in [1 . Thus, the number 
of SYT of truncated shifted staircase shape [m + 4] \ (1) is 

(m+3,m+l,...,l) (m+3,m+l) N\(2m + 4)! 

y y ' (7V-2m-5)!(4m + 9)! 

N\ ■ 4(2m + 3) jj 1 i\ 



(4m + 9)! • (m + 3) 11 (2i + 1)! ' 



where N = ( m + 5 ) - 1 = (m + 3) (777 + 6)/2 is the size of the shape. For 
(N = 9) this number is 4, as shown in Example [ 



5. Truncated Rectangular Shapes 

In this section, A = (Ai, . . . , A m ) (with Ai > . . . > A m > integers) will be 
a partition with (at most) m parts. Two partitions which differ only in trailing 
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zeros will be considered equal. Denote by f x the number of SYT of regular 
(non-shifted) shape A. 

For any nonnegative integers to and n, let (n m ) := (n, . . . , n) (to times) be 
the corresponding rectangular shape. The Frobenius- Young formula (Proposi- 
tion [2J} implies the following. 

Observation 5.1. The number of SYT of rectangular shape (n m ) is 

f^ n i = [mny ■ 



where 

F m := l[ il 

i=Q 

Recall from Section [2] the definition of the sum A + // of two partitions A and 
/i. Note that if either A or [i is a strict partition then so is A + /j. 

Lemma 5.2. Let m, n and t be nonnegative integers, with t < mn. Let T be a 
SYT of rectangular shape (n m ), let T\ be the set of all cells in T with values at 
most t, and let T2 be obtained from T\T\ by transposing the shape and replacing 
each entry i by mn — i + 1 . Then: 

(i) T\ and Ti are SYT. 

(ii) Denote by A 1 and A 2 the shapes ofT\ andT2, respectively, and treat strict 
partitions as sets. Then the strict partition [m + n] is the disjoint union 
of the strict partitions A 1 + [m] and A 2 + [n] . 

Proof, (i) is clear; let us prove (ii). The borderline between T\ and T\T\ is a 
lattice path of length exactly m + n, starting at the NE corner of the rectangular 
shape (n m ), using only S and W steps, and ending at the SW corner of this shape. 
If the first step is S then the first part of A 1 + [m] is m + n, and the rest (of both 
A 1 + [m] and A 2 + [n]) corresponds to a lattice path in n m ~ x . Similarly, if the 
first step is W then the first part of A 2 + [n] is m + n, and the rest corresponds 
to a lattice path in (n — l) m . Thus exactly one of the strict partitions A 1 + [m] 
and A 2 + [n] has a part equal tom + n, and the whole result follows by induction 
on to + n. 

□ 

Corollary 5.3. For any nonnegative integers m, n and t with t < mn, 



AC( n ™; 
tM=t 



Here summation is over all partitions A with the prescribed restrictions, and X c is 
such that X c + [n] is the complement of A + [to] in [to + n] ( where strict partitions 
are treated as sets). Ln particular, the LHS is independent oft. 
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Lemma 5.4. Let A and A c be partitions such that [m + n] is the disjoint union 
of A + [m] and A c + [n], and let fi = (jj,± } . . . , jik) be an arbitrary partition (fi± > 
...>Hk>0)' Let f x := f x /\\\\. Then 

j(ti+{n k ))U\ j(n+{m k ))\J\ a _ jX JX" ^+(m+n) k Jfi 

Equivalently, 

/ (^+(n fe ))UA / ( / ,+(m fe ))UA= = ^ ^ ^cp . f X f X^ 

where 

M„ \\\ l\ c h- fM+((«+") fc ) ( ImI + ^ + I ^| ) ! ( |/i| + mfc + | A C | ) ! 

M U ' ' ■(| M | + (m + n)fc)!(|M|)!(|A|)!(|Al)!- 

Proof. From the assumptions it follows that A is contained in (n m ). We may 
thus assume that it has m (nonnegative) parts. Similarly, A c is contained in (m n ) 
and we may assume that it has n (nonnegative) parts. Thus ((/x + (n k )) U A has 
k + m parts and (/i + (m k )) U A c has k + n parts. By Proposition ^. 11 

f(p+(n k ))UX f(p+(m k ))UX c 

' k i 

n (Mi+m+n+fc _ i) , n^-Mi-»+i) 

-i ^ 7 t<j 

• JJ(a*, + n - - i + fc + j) JJ(jUi + m — — « + + i) . 
By the assumption on A and A c , 

JJ^+n-Aj—i+fe+j) JJ^+m-AI-i+A+j) = JJ i+A+j) (Vi). 

3 j 3 = 1 

Since 

^ m+n _^ 



J]( M ,-i + fc + j)= (Vi) 



(/ii + m + n + fc — i)\ j-J^ (fj,i + k — i)\ 

an application of Proposition 12.11 to f>* and to f^+(( m + n ) k ) gives the desired 
result. 

□ 

Theorem 5.5. Let m and n be nonnegative integers, and let \x = (pi, . . . ,/ifc) 
be a partition (fj,i > . . . > fit > OJ. TTien 

j(p+(n fc ))UA j(p+(m fc ))UA c = y ?/1 +((m+n) fc ) j(™ m ) . 

AC(n m ) 

'mn + 2|/i| + mfc + n/c + lA (|/x| + mfc)!(|/z| + nfc)! 



mn J (\fi\ + mk + nk)\(\i/i\)\ 
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Proof. Restrict the summation to partitions A of a fixed size |A| = t. By 
Lemma [53] and Corollary 15.31 



XC(n™) AC(„'") 

ix|=t Ul=t 



Now sum over all t and use the explicit formula for d(/j,, t, M—t) (from Lemma l5~Il) 
together with Lemma \A. 51 to obtain the explicit formula above. 

□ 

Again, we shall apply this theorem in several special cases. In each case the 
result is a special case of equation (@} with appropriately chosen £ of one or 
the other of the shapes illustrated in diagram @. Note that, by Lemma [5.21 

v = {{n m )/xy = X c . 

First, let a = (n k ), (3 = (m k ) and /1 = (0 k ) (the empty partition with k 
"parts" ) . This corresponds to truncating a k x k square from the NE corner of a 
rectangular shape ((n + k) m+k ), but adding back the SW corner of this square; 
see the first shape in diagram ©. 

Corollary 5.6. The number of SYT of truncated rectangular shape ((n+k) m+k )\ 
(k k - 1 ,k- 1) is 

f{(m+n) k ) j(n m ) fmn + mk + nk + l\ (mfc)!(nfc)! _ N\(mk)\(nk)\ _ F m F n F k 
\ mn J (mk + nk)\ (mk + nk + 1)1 F m + n +fe ' 

where N = (to + k)(n + k) — k 2 + 1 = mn + mk + nk + 1 is the size of the shape 
and F n is as in Observation \5. 1{ 



For k = 1 we obtain 

j(n+i) m+1 _ N\m\n\ F m F n _ ^ F m+1 F n+1 
(m + n + 1)! F m+n+1 ' F m+n+2 



in accordance with Observation 15. II 

For k = 2 we obtain that the number of SYT of truncated rectangular shape 
((n + 2) m+2 )\(2,l) is 

jV!(2TO)!(2n)! _ F m F n 
(2TO + 2n + l)! ' F m+n+2 ' 
where N = (to + 2)(n + 2) — 3 = mn + 2m + 2n + 1 is the size of the shape. 

Now take a = (n fe ), j3 = (m k ) and /i = (l* -1 ^), for k > 2. This corresponds 
to truncating a (k — 1) x (k — 1) square from the NE corner of a rectangular shape 
((n + k) m+k ); see the second shape in diagram @. 

Corollary 5.7. The number of SYT of truncated rectangular shape ((n+k) m+k )\ 
((k - l) fc_1 ) is 

f {{m+n+i) k -\m+n) f (n m ).( mn + mk + nk + 2k - 1\ (mfc + fc-l)!(nfc + fc-l)! 

V mn )' (mk + nk + k - l)\(k - 1)\ 
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_ N\{mk + k- l)\(nk +k- l)!(m + n + l)\k F m F n F k _ 1 
(mk + nk + 2k - 1)! F m+n+ k+i ' 

where N = (m + k) (n + k) — (k — l) 2 = mn + mk + nk + 2k — 1 is the size of the 
shape and F n is as in Observation \5.1\ 

In particular, letting k = 2 and \x = (1,0) implies that the number of SYT of 
truncated rectangular shape ((n + 2) m+2 ) \ (1) is 

N\(2m+ l)!(2n+ 1)! • 2 F m F n 

(2m + 2n + 3)!( TYl + Tl + 2) -Fm+n+2 

where = (to + 2)(n + 2) — 1 = tott, + 2to + 2n + 3 is the size of the shape and 
F n is as in Observation l5.ll 

6. Final Remarks and Open Problems 

Most recently, the numbers of SYT of some other truncated shapes have been 
shown to have nice product formulas. Using complex analysis and volume com- 
putations, G. Panova proved such a formula for a rectangular shape minus a 
staircase in the NE corner [7]. Computer experiments indicate that there are 
other shapes with a similar property. 

The number of SYT of a rectangular shape (n m ) (n ^ to), with two cells 
truncated from the NE corner, seems in general to have large prime factors. 
Nevertheless, the situation is different for n — m. 

Conjecture 6.1. For n > 2 

f („")\(2) = (n 2 -2)!(3n-4)P-6 F^ 2 
1 (6n-8)!(2n-2)!(n-2)! 2 F 2 „V 

where 



F m := J] 



m — 1 

il 

i=0 



This empirical formula has been verified for n < 13. 

On the basis of further computations we conjecture that, if either n = (fc fe_1 ) 
or k = (k k ~ 2 ,k — 1) then, for n large enough, both /(™"A K anc | gl n ]\ K have no 
prime factor larger than the size of the shape. 

Question 6.2. 

(i) Find and characterize the truncated (and other non "unusual") shapes 
for which the number of SYT has no prime factor larger than the size of 
the shape. 

(ii) Give explicit product formulas for the number of SYT of such shapes. 



Acknowledgements: Many thanks to Amitai Regev and Richard Stanley for 
stimulating comments and references. Thanks also to an anonymous referee for 
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helpful comments. Extensive computer experiments have been conducted using 
the Maple and Sage computer algebra systems. 

This paper addresses a question presented by the authors in the Workshop on 
Combinatorics held at the Weizmann Institute in June 2010, in honor of Doron 
Zeilberger's 60 th birthday. Doron, thanks for many years of inspiration! 
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